Sensing system, work system, augmented-reality-image displaying method, augmented-reality-image storing method, and program

ABSTRACT

A sensing system with a detecting device that is used to detect a position of a target and a controller, where, for display on a display device or projection by a projection apparatus, the controller creates an augmented-reality image that shows: at least one of a setting related to detection of the target using the detecting device, a setting of a moving apparatus, and a setting of a robot that performs work on the target, a position of the target being recognized by the controller, a result of the detection of the target, a work plan of the moving apparatus, a work plan of the robot, a determination result of the controller and a parameter related to the target.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/451,668, filed on Jun. 25, 2019, which is based on and claimspriority to Japanese Patent Application No. 2018-139198, filed on Jul.25, 2018, the entire content of which is incorporated herein byreference.

FIELD

The present invention relates to a sensing system, a work system, anaugmented-reality-image displaying method, an augmented-reality-imagestoring method, and programs therefor.

BACKGROUND

In the related art, there is a known work system provided with a visualsensor for detecting the position of a target object being moved by amoving apparatus and a robot that performs picking work of the targetobject on the basis of the detection results of the visual sensor (forexample, see Japanese Unexamined Patent Application, Publication No.2005-111607).

In addition, there is a known system that displays the path of a taughtmotion of a robot on a display device on the basis of a 3D model of therobot and the position of the viewpoint of an operator on 3D coordinates(for example, see Japanese Unexamined Patent Application, PublicationNo. 2017-100234).

SUMMARY

A sensing system according to a first aspect of the present invention isprovided with: a detecting device that is used to detect at least aposition of a target that is being moved by a moving means for movingthe target; and a controller, wherein the controller creates, fordisplay on a display device or projection by means of a projectionapparatus, an augmented-reality image that shows at least one type ofthe following information: at least one of a setting related todetection of the target performed by using the detecting device, asetting of the moving means, a setting of a work machine that performswork on the target, which are settings being recognized by thecontroller; a position of the target being recognized by the controllerbased one basis of the settings; a result of the detection of the targetperformed by using the detecting device, a work plan of the movingmeans, a work plan of the work machine, a determination result of thecontroller related to execution of the work performed by the workmachine, and a parameter related to the target being recognized by thecontroller.

A sensing system according to a second aspect of the present inventionis provided with: a detecting device that is used to detect at least aposition of a target that is being moved by a predetermined movingmeans; a controller that controls a work machine that performs work onthe target by using detection result of the detecting device; anaugmented-reality-image creating unit which successively creates, whenthe work machine is performing the work, augmented-reality images thatshow at least one type of the following information: the position of thetarget being recognized by the controller on the basis of the detectionresult of the detecting device, a work plan of the moving means, a workplan of the work machine, a determination result of the controllerrelated to execution of the work performed by the work machine, and aparameter related to the target being recognized by the controller; anda storage unit that stores the augmented-reality images created by theaugmented-reality-image creating unit together with correspondingreal-space images.

A sensing system according to a third aspect of the present invention isprovided with: a detecting device that is used to detect a plurality oftargets that are being moved by a predetermined moving means; a targetdetecting unit which applies image processing for detecting the targetsto the image data obtained by the detecting device; anaugmented-reality-image creating unit which creates for a candidate in aplurality of candidates that appear in the processed image to which theimage processing has been applied, an augmented-reality image thatcorresponds to the candidate in which a predetermined parameter is equalto or greater than a detection threshold, and creates a candidateaugmented-reality image that corresponds to a candidate in the pluralityof candidates, in which the predetermined parameter is less than thedetection threshold and equal to or greater than a detection candidatethreshold; and a display device or a projection apparatus that displaysor projects the augmented-reality image and the candidateaugmented-reality image.

A work system according to a fourth aspect of the present invention isprovided with the sensing system and a work machine.

A fifth aspect of the present invention is an augmented-reality-imagedisplaying method that is employed in a system in which a controllercontrols a work machine or a predetermined apparatus based on adetection result of a target that is being moved by a moving means formoving the target, the augmented-reality-image displaying methodincluding: creating by the controller an augmented-reality image thatshows at least one type of the following information: at least one of asetting related to detection of the target, a setting of the movingmeans, a setting of a work machine, which are settings being recognizedby the controller; a position of the target being recognized by thecontroller based on the settings; a result of the detection of thetarget performed by using the detecting device; a work plan of themoving means; a work plan of the work machine; a determination result ofthe controller related to execution of the work performed by the workmachine; and a parameter related to the target being recognized by thecontroller, and the method further including: displaying the createdaugmented-reality image on a display device or projecting theaugmented-reality image using a projection apparatus.

A sixth aspect of the present invention is an augmented-reality-imagestoring method that is employed in a system in which a controllercontrols a work machine based on a detection result of a target that isbeing moved by a moving means for moving the target, theaugmented-reality-image storing method including: successively creating,when the work machine is performing the work, augmented-reality imagesthat show at least one type of the following information: a position ofthe target being recognized by the controller based on the detectionresult; a work plan of the moving means; a work plan of the workmachine; a determination result of the controller related to executionof the work performed by the work machine; and a parameter related tothe target being recognized by the controller, and the method furtherincluding: storing the augmented-reality images created in theaugmented-reality-image creating step together with correspondingreal-space images.

A seventh aspect of the present invention is an augmented-reality-imagedisplaying method that is employed in a system that performs imageprocessing for detecting a plurality of targets that are being moved bya moving means for moving the targets, the augmented-reality-imagedisplaying method including: creating, for a candidate in a plurality ofcandidates that appear in the processed image to which the imageprocessing has been applied, an augmented-reality image that correspondsto the candidate in which a predetermined parameter is equal to orgreater than a detection threshold, and creating a candidateaugmented-reality image that corresponds to a candidate in the pluralityof candidates, in which the predetermined parameter is less than thedetection threshold and equal to or greater than a detection candidatethreshold; and displaying the augmented-reality image and the candidateaugmented-reality image.

An eighth aspect of the present invention is a program that is employedin a system in which a controller controls a work machine or apredetermined apparatus based on a detection result of a target that isbeing moved by a moving means for moving the target, the storage mediumstoring a program which causes a computer to execute: creating anaugmented-reality image that shows at least one type of the followinginformation: at least one of a setting related to the detection of thetarget, a setting of the moving means, a setting of a work machine,which are settings being recognized by the controller; a position of thetarget being recognized by the controller based on the settings; aresult of the detection of the target performed by using the detectingdevice; a work plan of the moving means; a work plan of the workmachine; a determination result of the controller related to executionof the work performed by the work machine; and a parameter related tothe target being recognized by the controller; and displaying thecreated augmented-reality image on a display device or projecting theaugmented-reality image using a projection apparatus.

A ninth aspect of the present invention is a program that is employed ina system in which a controller controls a work machine on the basis of aresult of detection of a target that is being moved by a moving meansfor moving the target, the storage medium storing a program which causesa computer to execute: successively creating, when the work machine isperforming the work, augmented-reality images that show at least onetype of the following information: a position of the target beingrecognized by the controller based on the detection result; a work planof the moving means; a work plan of the work machine; a determinationresult of the controller related to execution of the work performed bythe work machine; and a parameter related to the target being recognizedby the controller; and storing the augmented-reality images created inthe augmented-reality-image creating step together with correspondingreal-space images.

A tenth aspect of the present invention is a program that is employed ina system that performs image processing for detecting a plurality oftargets that are being moved by a moving means for moving the targets,the storage medium storing a program which causes a computer to execute:creating, for a candidate in a plurality of candidates that appear inthe processed image to which the image processing has been applied, anaugmented-reality image that corresponds to the candidate in which apredetermined parameter is equal to or greater than a detectionthreshold, and creating a candidate augmented-reality image thatcorresponds to a candidate in the plurality of candidates, in which thepredetermined parameter is less than the detection threshold and equalto or greater than a detection candidate threshold; and a displayingstep of displaying the augmented-reality image and the candidateaugmented-reality image.

BRIEF DESCRIPTION OF THE DRWAINGS

FIG. 1 is a diagram showing an schematic configuration of a sensingsystem according to an embodiment of the present invention and anexample movement thereof.

FIG. 2 is a block diagram of a controller of the sensing system of thisembodiment.

FIG. 3 is a diagram showing an example display image of a display deviceof the sensing system of this embodiment.

FIG. 4 is a diagram showing an example display image of the displaydevice of the sensing system of this embodiment.

FIG. 5 is a diagram showing an example display image of the displaydevice of the sensing system of this embodiment.

DETAILED DESCRIPTION

A sensing system according to an embodiment of the present inventionwill be described below with reference to the drawings.

As shown in FIG. 1, a sensing system according to this embodiment isprovided with: a moving apparatus 10 that serves as a moving means fortransferring a target object O as an object; a detecting device 20 thatis used to detect the target object O that is moved by the movingapparatus 10; a robot 30 which is a work machine that performspredetermined work on the target object O being moved by the movingapparatus 10; a controller 40 that controls the robot 30; and anaugmented reality apparatus 60 (FIG. 2). The sensing system of thisembodiment is also a work system because said sensing system is a systemin which the robot 30, which is a work machine, performs thepredetermined work on the sensed target object O.

The moving apparatus 10 moves the target object O as a result of drivingthe motor 11. The motor 11 is provided with an operating-positiondetecting device 12, and the operating-position detecting device 12successively detects the rotational position and the rotation amount ofan output shaft of the motor 11. The operating-position detecting device12 is, for example, an encoder, and values detected by theoperating-position detecting device 12 are transmitted to the controller40.

Note that the operating position of the target object O may be detectedby the detecting device 20 without using an encoder.

The target object O may be any of various types of articles, such as avehicle body, a frame, a component, a food product, and a medicalproduct. It is possible to employ a moving apparatus other than themoving apparatus 10 shown in FIG. 1, and it suffices so long as themoving apparatus 10 is capable of transferring the target object O.

The detecting device 20 is a 2D camera, a 3D camera, a 3D distancesensor, or the like, and the detecting device 20 transmits obtainedimage data (detection results) or detected points (detection results) ofthe 3D distance sensor to the controller 40. As will be described later,the controller 40 detects the position and orientation of the targetobject O on the basis of the image data or the detected points.

Although the robot 30 is not limited to a specific type of robot, therobot 30 of this embodiment is an articulated robot provided with aplurality of servomotors 31 that individually drive a plurality ofmovable portions (see FIG. 2). Note that an arm 30 a of the robot 30 isconstituted by the plurality of movable portions. The servomotors 31have operating-position detecting devices, respectively, for detectingthe operating positions thereof, and the operating-position detectingdevices are, for example, encoders. The values detected by theoperating-position detecting devices are transmitted to the controller40. By doing so, the controller 40 can recognize the position, velocity,acceleration, and so forth of each portion of the robot 30.

The robot 30 performs predetermined work on the target object O inaccordance with the type of a tool 50 provided at a tip portion thereof.It suffices so long as the tool 50 is a tool for a robot, including agripping tool, a vacuum tool, a processing tool, a cleaning tool, anassembly tool, a measuring tool, various types of sensors, avision-system camera, or the like. A processing tool is, for example, adrilling tool such as an electric drill, a threading tool provided witha tap at the tip thereof, an electric lapping tool, a painting tool suchas a paint gun, or a welding tool such as a servo gun. A tool employedin assembly is, for example, an electric driver or a tool for gripping apin and inserting the pin into a hole. A measuring tool is, for example,a film-thickness measuring device, an ultrasonic internal inspectionapparatus, a durometer, a non-contact thermometer, or a close-up camera.Note that the target object O may be a hole provided in an object, aprocessing-target portion, or the like.

The tool 50 of this embodiment is a hand, and the robot 30 unloads thetarget object O on the moving apparatus 10 by using the tool 50.

The tool 50 is provided with a servomotor 51 that drives claws (see FIG.2). The servomotor 51 has an operating-position detecting device fordetecting the operating position thereof, and an example of theoperating-position detecting device is an encoder. The values detectedby the operating-position detecting device are transmitted to thecontroller 40.

The coordinate system of the robot 30 and the coordinate system of themoving apparatus 10 are associated with each other. In this embodiment,as a result of bringing a setting tool provided at the tip portion ofthe robot 30 into contact with, at a plurality of predeterminedlocations, calibration jig installed on the moving apparatus 10, areference coordinate system to be used as the coordinate system of therobot 30 and the moving apparatus 10 are set. In addition, on the basisof image data of the calibration jig obtained by the detecting device20, the position and orientation (sensor coordinate system) of thedetecting device 20 are associated with the reference coordinate system.

As shown in FIG. 2, the controller 40 is provided with: a control unit41 having a CPU, a RAM, and so forth; a controller display unit 42 thatdisplays the setting, menu, and so forth of the controller 40; a storageunit 43 having a non-volatile storage, a ROM, and so forth; a pluralityof servo controllers 44 that individually correspond to the servomotors31 of the robot 30; a servo controller 45 that corresponds to theservomotor 51 of the tool 50; and an input unit 46 connected to thecontroller 40. The input unit 46 is an input device such as an operationpanel that an operator can carry. There are cases in which the inputunit 46 wirelessly communicates with the controller 40.

The storage unit 43 stores a system program 43 a, and the system program43 a carries out the basic functions of the controller 40. In addition,the storage unit 43 stores a motion program 43 b, a target-objectdetecting program 43 c, and so forth. On the basis of the target-objectdetecting program 43 c, the control unit 41 applies, to the image datatransmitted thereto from the detecting device 20, well-known imageprocessing, center-of-gravity-position detecting processing, patternmatching processing, other necessary processing, and so forth to detectthe target object O, and detects the position, the orientation, and soforth of the target object O being moved by the moving apparatus 10.Note that, in the case in which the detecting device 20 is capable ofperforming the image processing, the detecting processing, the patternmatching processing, and so forth, the control unit 41 does not need toperform processing that is redundant with the processing performed bythe detecting device 20.

In addition, on the basis of the target-object detecting program 43 c,the control unit 41 follows (tracks) at least the position andorientation of the target object O being moved by the moving apparatus10, starting from immediately after the timing when the target object Ois detected, by using the detection results of the operating-positiondetecting device 12 (the detection results of the movement of the movingapparatus 10). In this way, on the basis of the image data transmittedthereto from the detecting device 20 and the detection results of theoperating-position detecting device 12, the control unit 41 is capableof detecting the position, the velocity, the movement amount, and soforth of the target object O being moved by the moving apparatus 10.

The augmented reality apparatus 60 includes: a control unit having aCPU, a RAM, and so forth; a storage unit having a non-volatile storage,a ROM, and so forth; and an input unit such as a keyboard, and so forth.As shown in FIG. 2, in an example, a camera 61 that is installed at apredetermined position in order to acquire images of the movingapparatus 10 and the robot 30 and a display device 62 that displaysreal-space images acquired by the camera 61 are provided. It is possibleto employ a publicly known display device, such as a tablet-type device,a head-mounted display, and glasses with an AR display function, as thedisplay device 62. In this case, the camera 61 may be installed in atablet-type device, a head-mounted display, or glasses with an ARdisplay function. The image data acquired by the camera 61 istransmitted to the controller 40. In addition, the display device 62displays the augmented-reality image data transmitted thereto from thecontroller 40. The augmented-reality images may be 3D images or 2Dimages. Hereinafter, “augmented reality” may also be indicated as “AR”.On the basis of the image data acquired by the camera 61 regarding marksaffixed to the calibration jig and the robot or the shape of the robotitself, the position and orientation of the camera 61 (camera coordinatesystem) may be associated with the reference coordinate system. In orderto associate the position of the real space and the positions of the ARimages without changing the dimensions thereof, it is preferable thatthe real-space images acquired by the camera 61 are 3D images acquiredby a 3D camera; however, it is also possible to employ another type ofcamera that acquires 2D images as real-space images.

Note that the reference coordinate system may automatically beassociated with the camera coordinate system, or the coordinate systemsmay be manually aligned by a user directly manipulating the AR images.

Note that the reference coordinate system need not be associated withthe camera coordinate system. In this case, although the AR images arenot superimposed on real space, the relative relationship of the objectsin real space and the relative relationship of the objects in the ARimages are identical, and thus, if there is an abnormality in therelative relationship of the objects in the AR images, it is understoodthat the actual setting is also not accurate.

(Basic Processing for Displaying Augmented Reality)

On the basis of an AR-image creating program 43 d stored in the storageunit 43, the control unit 41 successively creates AR image data relatedto at least one type of the following information pieces: the settingsrelated to the detection of the target object O performed by using thedetecting device 20; the setting of the moving apparatus 10; the settingof the robot 30; the position and orientation of the target object Obeing recognized by the control unit 41 on the basis of the settings;the result of the detection of the target object O performed by thedetecting device 20; the work plan of the moving apparatus 10; the workplan of the robot 30; the determination result of the control unit 41related to the execution of the work performed by the robot 30; and theparameters related to the target object O being recognized by thecontrol unit 41, and the control unit 41 transmits the created AR imagedata to the display device 62. The positions and orientations of the ARimages are adjusted on the basis of the camera coordinate system or thereference coordinate system. By doing so, the display device 62 displaysthe real-space images received from the camera 61 and the AR imagesreceived from the controller 40.

Note that the AR images may be created by the augmented realityapparatus 60 by acquiring the data from the control unit 41.

The communication between the controller 40 and the augmented realityapparatus 60 may be performed via wires or may be performed wirelessly.

In the example in FIG. 1, as the settings related to the detection ofthe target object O performed by using the detecting device 20, an ARimage 71 of the sensor coordinate system of the detecting device 20 andan AR image 72 of the detection area of the detecting device 20 aredisplayed. As the settings related to the detection of the target objectO performed by using the detecting device 20, AR images indicating theposition and direction of the detecting device 20 may be displayed, andthe frame rate of the detecting device 20, the position and direction ofthe detecting device 20, the adjustment values of the aperture and theexposure time of the detecting device 20, the elapsed time after thedetecting device 20 was firstly used in operation, and so forth may bedisplayed in the forms of AR images of indicators, numerical values,characters, and so forth. The indicators may express the magnitudes bymeans of colors.

The position and area of the AR image 72 of the detection area are seton the basis of calibration. For these settings, for example, theposition and size of a mark, a pattern, the target object O, or the likeon the moving apparatus 10 may be employed as references. In the case inwhich the AR image 72 of the detection area is displayed, it is possiblefor the operator to easily check the relationship between the detectionarea and work area of the robot 30 in order to see whether or not theplacement of the target object O on the moving apparatus 10 isappropriate.

In the example in FIG. 1, on the basis of the positions and orientationsof the target objects O detected by the control unit 41, AR images 73that indicate the positions and orientations of the target objects Obeing recognized by the control unit 41 are displayed. FIG. 1 shows anexample in which the AR images 73 are deviated from the target objects Oin the real-space images in a width direction of the moving apparatus10. In this embodiment, although the AR images 73 have shapes that aresimilar to or identical to the shapes of the target objects O, the ARimages 73 may be points that indicate the positions, orientations, andso forth of the target objects O, a display image that indicates themovement directions, velocities, and so forth of the target objects O,abstract views of the target objects O, other alternative diagrams, orthe like. Similar display image is possible for other AR imagesdescribed above and described below. The positions and orientations ofthe AR images 73 change in accordance with the positions andorientations of the target objects O detected by the control unit 41.

In addition, in the example in FIG. 1, the reference coordinate system,which is a setting of the robot 30, is displayed as an AR image 74. Asthe setting of the robot 30, the real motion velocity of the robot 30,the maximum velocity of the robot 30, the elapsed time after the robot30 was firstly used in operation, and so forth may be displayed as ARimages of indicators, numerical values, characters, and so forth.

As a result of displaying the calculation results of the velocities oraccelerations of the tip portion of the robot 30, the tool 50, thetarget object O gripped by the tool 50, and so forth in the forms of ARimages of indicators, numerical values, characters, and so forth, it ispossible to intuitively recognize the suitability or the like of theload applied to the tool 50, the load applied to the target object Ogripped by the tool 50, the gripping force of the tool 50, and so forth.The velocities and the accelerations may reflect the rotation of a wristflange of the robot 30.

In addition, in the example in FIG. 1, the work area of the robot 30,which is a setting of the robot 30, is displayed as an AR image 75. Inthis embodiment, the work area is an area in which the robot 30 performspicking work of the target objects O. With this configuration, it ispossible for the operator to intuitively recognize the relationshipbetween the work area and an area in which the operator operates, therelationship between the work area and the work area of another robot,the size of the work area with respect to the work contents, and soforth.

In addition, in the example in FIG. 1, as the setting of the movingapparatus 10, an AR image 76 of the transfer direction of the movingapparatus 10 is displayed. As the setting of the moving apparatus 10,the transfer velocity, the elapsed time after the moving apparatus 10was firstly used in operation, and so forth may be displayed in theforms of AR images of indicators, numerical values, characters, and soforth.

The transfer direction of the moving apparatus 10 is, for example,calculated on the basis of positional changes of the target objects Obeing moved by the moving apparatus 10. In this case, it is preferablethat the camera 61 be provided above the moving apparatus 10. Thisconfiguration makes it possible to visually check the relationshipbetween the AR image 74 and the AR image 76, which is advantageous forchecking whether or not the transfer direction of the moving apparatus10 is appropriate with respect to the setting of the referencecoordinate system or the like.

In addition, as the setting of the moving apparatus 10, an indicationmark that points at a specific position where a belt of the movingapparatus 10 being recognized by the controller 40 is located may bedisplayed as an AR image 77 in order to indicate the transfer velocityof the moving apparatus 10.

In addition, a motion restricted region of the robot 30 may be displayedon the display device 62 as an AR image. The motion restricted region isa region that is set in an area surrounding the operator, peripheralequipment, or the like, and is a region in which the motion of the robot30 is halted or restricted. In this case, it becomes possible tointuitively recognize the area set to be the motion restricted region.

In addition, as the results of the detections of the target objects O bythe detecting device 20, images detected by the detecting device 20 orimages obtained by applying the image processing to said images,representing the positions, orientations, shapes, or the like of thetarget objects O may be displayed on the display device 62 in the formof AR images.

In this case, it is possible for the operator to intuitively recognizethe suitability of the detections of the target objects O by thedetecting device 20. For example, it is possible to check thesuitability of lighting on the target objects O, the suitability of adetection threshold with respect to the lighting on the target objectsO, the suitability of the positions, the orientations, the shapes, orthe like of the target objects O that have been detected by thedetecting device 20 and that are being recognized by the controller 40.In the case in which the lighting and the detection threshold are notappropriate, a portion of or the entirety of a target object O that is abright portion or a dark portion may not be detected, which tends tooccur frequently. In addition, by comparing the positions, orientations,and shapes of the target objects O detected by the detecting device 20with those of the actual objects, the operator can intuitively check thesuitability of the detections by the detecting device 20. Thisconfiguration is useful because the detection by the detecting device 20greatly influences the success/failure of the work performed by therobot 30, the suitability thereof, and so forth.

Note that it is also possible to successively update, on the basis ofthe results of the detections of the target objects by the detectingdevice 20, the positions, orientations, and shapes of the target objectsO, which are displayed on the display device 62 in the form of ARimages, in accordance with the actual movement of the target objects.

In addition, at an arbitrary point in time, it is possible to stopupdating and to check the results, and it is also possible to replay therecorded results.

In addition, as shown in FIG. 3, as the work plan of the robot 30,indicators, numerical values, characters, and so forth that indicate aplanned order in which the robot 30 performs work on a plurality oftarget objects O that are followed by the control unit 41, whether ornot it is possible to perform the work, the work difficulty (anattribute of the target objects O), the type, weight, hardness, andmaterial of the target objects O (attributes of the target objects O),inspection results of the target objects O, and so forth may bedisplayed on the display device 62 in the form of AR images 78. Here,the planned order, whether or not it is possible to perform the work,and the work difficulty are determination results of the control unit 41related to the execution of the work, and the type, weight, hardness,material, and inspection result of the target objects O are theparameters related to the target objects O being recognized by thecontrol unit 41.

Note that it is also possible to successively update, on the basis ofthe results of the detections of the target objects by the detectingdevice 20, the AR images 78 displayed on the display device 62 inaccordance with the actual movement of the target objects. In addition,at an arbitrary point in time, it is possible to stop updating and tocheck the results, and it is also possible to replay the recordedresults.

In this case, it is possible for the operator to intuitively recognizethe work plan of the robot 30. For example, as shown in FIG. 3, in thecase in which the planned order of a target object O on the downstreamside in the transfer direction is not appropriate among the plurality oftarget objects O disposed between the pair of AR images 75, whichindicate the work area of the robot 30, it is possible to easilyrecognize this fact.

Note that, as shown in FIG. 4, in the case in which a plurality ofrobots 30 perform work on the target objects O, it is possible todisplay, on the display device 62 in the form of AR images 79, theplanned order with which the work is performed for the respective robots30. This configuration makes it possible to intuitively recognize thedistribution of work among the individual robots 30 or the like. The ARimages 79 of the work order may be individually transmitted to thedisplay device 62 from the controllers 40 of both robots 30, or the ARimages 79 may be transmitted to the display device 62 from thecontroller 40 of one of the robots 30. In addition, the AR images 79 ofthe work order may be transmitted to the display device 62 from ahigher-level controller that is connected to the robots 30. Here, theplanned order is a determination result of the control unit 41 relatedto the execution of the work.

Note that it is also possible to successively update, on the basis ofthe results of the detection of the target objects by the detectingdevice 20, the AR images 79 displayed on the display device 62 inaccordance with the actual movement of the target objects. In addition,at an arbitrary point in time, it is possible to stop updating and tocheck the results, and it is also possible to replay the recordedresults.

In addition, as the work plan of the moving apparatus 10, it is alsopossible to display the future positions and future orientations thatare expected in the near future for the plurality of target objects Obeing followed by the control unit 41 on the display device 62 as ARimages. In this case, the operator can easily check the movementdirections, orientations during movement, and so forth of the targetobjects O.

Note that it is also possible to successively update, on the basis ofthe results of the detections of the target objects by the detectingdevice 20, the AR images displayed on the display device 62 inaccordance with the actual movement of the target objects. In addition,at an arbitrary point in time, it is possible to stop updating and checkthe results, and it is also possible to replay the recorded results.

As a result of displaying the AR images on the display device 62 as hasbeen described above, it is possible for the operator to intuitivelyrecognize the settings related to the detections of the target objects Operformed by using the detecting device 20, the settings of the movingapparatus 10, the settings of the robot 30, the positions andorientations of the target objects O being recognized by the controlunit 41 on the basis of the settings, the results of the detections ofthe target objects O by the detecting device 20, the work plan of themoving apparatus 10, the work plan of the robot 30, and thedetermination results of the control unit 41 related to the execution ofthe work performed by the robot 30, or the parameters related to thetarget objects O being recognized by the control unit 41. This effect isextremely advantageous in order: to reduce the amount of work timerequired to perform the settings, grasp a phenomenon, investigate thecause of the phenomenon, and so forth; to enhance the work efficiencyvia improvement of settings; and to enhance the sense of setting.

(Displaying Detection Results that do not Qualify as Work Targets)

In FIGS. 3 and 4, the individual AR images 73 show the target objects Oas work targets on which the robot(s) 30 can perform work. In contrast,for example, when detections of the target objects O are notsatisfactory, when the orientations of the detected target objects O arenot appropriate for performing the work thereon, or the like, there arecases in which the robot(s) 30 do not execute or cannot execute the workon some of the target objects O among the plurality of target objects O.For example, when a detection parameter of a target object O is lessthan a detection threshold thereof, this target object O would not bedetected as a work target. In addition, when a parameter related to theorientation of a target object O in a detected image is less than anorientation threshold thereof, it is determined that the orientation ofthe target object O is not appropriate for performing the work thereon.

The lighting on the target objects O, the orientations of the targetobjects O, the shapes of the target objects O, the material of thetarget objects O, the settings of the detecting device 20 including theaperture, the exposure time, and so forth are the causal factors thatinfluence appropriate detections of the target objects O, and thesettings of the detection thresholds employed in determination ofwhether or not a target object O will be detected as a work target alsoinfluence appropriate detections of the target objects O. As thedetection thresholds, it is possible to use a parameter related tocontrast, a parameter that indicates the difference or similarity of theshape with respect to a model, or the like.

The control unit 41 creates the AR images 73 that correspond to, amongcandidates that respectively correspond to the target objects O, foreignobjects, blemishes on the moving apparatus 10, and so forth in the imagedata to which the image processing has been applied, candidates in whicha predetermined parameter is equal to or greater than the detectionthreshold, and creates AR images (candidate AR images) 80 thatrespectively correspond to candidates in which the predeterminedparameter is less than the detection threshold and the predeterminedparameter is equal to or greater than a detection candidate threshold(FIG. 5). Here, the detection candidate threshold is less than thedetection threshold. The AR images 80 are created in a display form thatmakes it possible to distinguish the AR images 80 from the AR images 73,and, accordingly, the AR images 80 are displayed on the display device62 in a form that makes it possible to distinguish the AR images 80 fromthe AR images 73.

With this configuration, it is possible for the operator to relativelyeasily perform the adjustment of the causal factors for increasing thenumber of target objects O that are detected at or above the detectionthreshold, the adjustment of the detection threshold, and so forth.

In addition, the control unit 41 may create AR images 81 of score valuesthat accompany the individual AR images 80, respectively. The scorevalues are indicators that indicate the similarity and degree ofconcordance between a model and target objects, and, in FIG. 5, thepossibility of reaching the detection threshold increases when the scorevalue increases. The score values are examples of detection results,and, as AR images of the detection results, AR images of contrastvalues, detected positions, type information of the target objects O, IDinformation of the target objects O, measurement results, and so forthmay be created. As a result of the AR images 81 being displayed on thedisplay device 62, it is possible for the operator to more easilyperform adjustment of the causal factors for increasing the number ofthe target objects O that are detected at or above the detectionthreshold, the adjustment of the detection threshold, and so forth.

Note that, AR images 71-81 are successively created and recorded(updated) as the time passes. Accordingly, the AR images 71-81 are alsosuccessively updated on the display device 62 in accordance with thereal-space images.

Alternatively, the controller 40 may create and record the AR images71-81 in real time, and the display device 62 may display AR imagesselected from the plurality of AR images 71-81 at that point in time orin the most recent past, in accordance with predetermined inputs to aninput unit or establishment of predetermined conditions in thecontroller 40 or the sensors.

(Re-Display Processing)

The controller 40, the augmented reality apparatus 60, and so forth havea function for saving the real-space images and the AR images 71-81displayed on the display device 62. In the case in which the controller40 saves the images, the image data from the camera 61 successivelyreceived from the augmented reality apparatus 60 and the AR image datathat are successively created are at least temporarily saved in thestorage unit 43 on the basis of a replaying program 43 e stored in thestorage unit 43. In the case in which the augmented reality apparatus 60saves the images, the image data from the camera 61 and the AR imagedata that are successively received from the controller 40 are at leasttemporarily saved in a storage unit of the augmented reality apparatus60 on the basis of a replaying program stored in the storage unit of theaugmented reality apparatus 60. Another apparatus, computer terminal,and so forth may similarly save the images.

Also, the controller 40 or the augmented reality apparatus 60 performs,in accordance with the operation of the input unit 46 or an input unitof the augmented reality apparatus 60, replay, slow replay, pauseddisplay during replay, and so forth on the controller display unit 42,the display device 62, and so forth by using the saved real-space-imagedata and AR image data. Accordingly, it is possible for the operator torepeatedly view the superimposed display of the real-space images andthe AR images 71-81.

With this configuration, it is possible for the operator to moreaccurately recognize the settings related to the detections of thetarget objects O performed by using the detecting device 20, thesettings of the moving apparatus 10, the settings of the robot 30, thepositions and orientations of the target objects O being recognized bythe control unit 41 of the controller 40 on the basis of the settings,the results of the detections of the target objects O by the detectingdevice 20, the work plan of the moving apparatus 10, or the work plan ofthe robot 30.

Note that it is also possible for the controller 40, the augmentedreality apparatus 60, or the like to transmit the saved real-space-imagedata and AR image data to another display device.

(Displaying Augmented-Reality Image in Accordance with Selection)

The augmented reality apparatus 60 has a function for displaying onlyselected AR images on the display device 62. For example, only the ARimages 71 selected from the AR images 71-81 are displayed on the displaydevice 62 in accordance with the operation of the input unit of theaugmented reality apparatus 60. In FIG. 4, it is also possible todisplay only the AR images related to one robot 30 among the pluralityof robots 30 on the display device 62.

With this configuration, it is possible for the operator to change theAR images to be displayed, as needed, and this is advantageous forreducing the amount of work time related to performing settings,ascertaining a phenomenon, investigating causes of the phenomenon, andso forth.

Effects that are similar to those described above are also achieved inthe case in which the controller 40 transmits only the selected ARimages to the display device 62.

In addition, the augmented reality apparatus 60 may have a function fordisplaying, on the display device 62, only the AR images in a selectedarea. For example, only the AR images that are present in a selectedarea are displayed on the display device 62 in accordance with theoperation of the input unit of the augmented reality apparatus 60.

In this case also, it is possible for the operator to change the area inwhich the AR images are displayed, as needed, and this is advantageousfor reducing the amount of work time related to performing settings,ascertaining a phenomenon, investigating causes of the phenomenon, andso forth.

Note that, in the case in which the display device 62 has a touch screenfunction, the selected area mentioned above could be a predeterminedarea centered on a finger, a pen, or the like that is brought intocontact with the display device 62 or an area in which the boundarythereof is drawn by using a finger, a pen, or the like.

Note that, in the case in which the display device 62 is a head-mounteddisplay or glasses with an AR function, the display device 62 may beprovided with a line-of-sight detecting device that detects the line ofsight of a wearer. In this case, it is possible for the line-of-sightdetecting device to serve as the input unit of the augmented realityapparatus 60.

(Displaying Augmented-Reality Image of Execution History in Accordancewith Selection)

The augmented reality apparatus 60 has a function for displaying anexecution history related to a selected AR image on the display device62. For example, an image of a target object O corresponding to one ARimage 73 or AR image 80 selected from the plurality of AR images 73 and80 acquired by the detecting device 20 may be displayed on the displaydevice 62 as the execution history in accordance with the operation ofthe input unit of the augmented reality apparatus 60. It is preferablethat the execution history be displayed at a location that does notoverlap with the AR images 73 and 80 on the display device 62.

Note that, when the image acquisition by the detecting device 20 and thedetecting processing by the control unit 41 are performed, a pluralityof candidates that could be target objects O are recognized by thecontrol unit 41. The control unit 41 is capable of saving executionhistories that respectively correspond to the candidates in the storageunit 43.

It suffices that the execution history is a result obtained as a resultof execution of the sensing system, and may be a measurement result, adetection result, image-acquisition result, or the like.

(Selection and Manipulation of Object)

In addition, the augmented reality apparatus 60, the controller 40, andso forth may perform selection processing of the individual AR images71-81 displayed on the display device 62 and movement processing of theselected AR images in accordance with an input made by using apredetermined input device.

For example, on the basis of an AR-image manipulating program 63 astored in a storage unit 63 of the augmented reality apparatus 60, theaugmented reality apparatus 60 sets one or more of the individual ARimages 71-81 to the selected state in accordance with an input made byusing the predetermined input device, and changes the positions andorientations of the selected AR images on the display device 62 inaccordance with the input made by using the predetermined input device.

When changing the positions and orientations of the AR images 73 of thedetected target objects O, it is possible to change only the positionsof the AR images 73 manipulated via the predetermined input device, andit is also possible to move the positions of the plurality of AR images73 by using the same movement distance or corresponding movementdistances.

The predetermined input device includes, for example, an input devicewhich is a touch screen, a keyboard, or the like provided in the displaydevice 62.

In addition, the augmented reality apparatus 60 transmits movementinformation of the AR images 71-81 to the controller 40 on the basis ofthe input made by using the input device, and the control unit 41 of thecontroller 40 changes, on the basis of a setting adjusting program 43 fstored in the storage unit 43, various types of parameters that are setin the controller 40 or various types of parameters that are calculatedin the controller 40, in accordance with the received movementinformation.

For example, in accordance with the movement information of the ARimages 71 or the AR images 74, the sensor coordinate system or thereference coordinate system being recognized by the controller 40 ischanged. In the case in which the detecting device 20 has a zoomfunction, an optical-axis changing function, an orientation changingfunction, and so forth for changing the detection area, the controller40 controls these functions of the detecting device 20 in accordancewith the movement information of the AR image 72.

In addition, detected positions, orientations, and so forth of some ofor all of the target objects O being recognized by the controller 40 arechanged in accordance with the movement information of the AR images 73or the AR images 80. The transfer velocity, the movement amount, and soforth of the individual target objects O being recognized by thecontroller 40 may be changed in accordance with the movement informationof the AR images 73 or the AR images 80. On the other hand, the transfervelocity, the movement amount, and so forth of the target objects Obeing recognized by the controller 40 may be changed in accordance withthe movement information of the AR image 77.

Note that respective data may be changed with respect to the recorded ARimages in accordance with respective items of the movement information.

In addition, the controller 40 changes the settings of the work area ofthe robot 30 in accordance with the movement information of the AR image75. In addition, the transfer direction of the moving apparatus 10 beingrecognized by the controller 40 is changed in accordance with themovement information of the AR image 76.

Note that, although the augmented reality apparatus 60 may transmit themovement information to the controller 40 each time the AR images aremoved, a configuration in which the movement information is transmittedon the basis of an input made by using the input device is moreadvantageous in terms of reducing the processing amount, setting error,and so forth of the controller 40.

Note that it is also possible for the control unit 41 to receive themovement information of a certain AR image, and to change, with regardto the AR images that are different from said AR image, various types ofparameters that are set in the controller 40 or various types ofparameters that are obtained by means of calculations performed in thecontroller 40. For example, the parameters of the AR images 73, 78, 79,80, and 81 are changed when the movement information of the AR images71, 74, and 75, and so forth related to the settings are received. Inthis case, on the basis of the AR-image creating program 43 d, thecontrol unit 41 recreates AR image data on the basis of the parametersthat have been changed in the controller 40, and transmits the createdAR image data to the display device 62. Accordingly, it is possible forthe operator to immediately recognize, when a certain AR image is moved,changes in the settings, calculations, and so forth in the controller 40associated with the movement of said AR image.

Note that it is also possible to employ a wearable device known as asmart glove as the predetermined input device. In the case in which asmart glove is employed, for example, when the operator places the handwearing the smart glove in the angle of view of the camera 61, thepositions of the index finger, the thumb, and so forth of said hand arerecognized by the augmented reality apparatus 60, and, when the operatorperforms the motion of grabbing one AR image among the AR images 71-81with the index finger and the thumb, said AR image is set to theselected state and is moved when the index finger and the thumb aremoved while being held in that state.

Note that the positions, orientations, parameters, and so forth of theabove-described AR images 71-81 are set, for example, by using thereference coordinate system of the robot 30 as the reference. Incontrast, the positions, orientations, parameters, and so forth of theAR images 71-81 may be set by using the sensor coordinate system of thedetecting device 20 as the reference. In addition, in the case in whichthe plurality of robots 30 individually have the controllers 40, as aresult of displaying the AR images 71-81 on the display device 62 inaccordance with the reference coordinate systems being recognized by therespective controllers 40, checking of the suitability of the settingsof the reference coordinate systems of the respective robots 30 isfacilitated.

In addition, in the case in which the display device 62 has a touchscreen function, a line-of-sight detecting function for detecting theline of sight of the operator, or the like, one or more of the pluralityof robots 30 may be specified by means of the pointing device, theline-of-sight direction, or the like, and the AR images being recognizedby the controller 40 of the specified robot 30 may be displayed.

(Application to Another Work System or Inspection System)

It is also possible to apply configurations which are the same as orsimilar to those described above to a work system in which the robot 30performs other types of work on the target objects O, such as cutting,assembly, inspection, and observation. It suffices that the targetobjects O are objects that can be moved by some kind of moving means,and it is possible to employ a robot that is different from the robot 30as the moving means. In the case in which the target objects O arevehicle bodies or frames of automobiles and in the case in which saidvehicle bodies or frames move by means of engines, motors, wheels, andso forth installed therein, the engines, the motors, the wheels, and soforth serve as the moving means.

The target objects O may be moved by means of a chute on which thetarget objects O slide down, roll down, or fall down by gravity. In thiscase, by causing an inclined chute to vibrate by means of a vibratingapparatus, it is possible to smoothly move the target objects O on thechute. In this case, the chute, the vibrating apparatus, and so forthserve as the moving means.

In addition, it is possible to apply configurations that are the same asor similar to those described above to a work system that does notemploy a robot, an inspection system, and so forth. Examples of a worksystem that does not employ a robot include an automatic painting systemin which a plurality of paint guns (predetermined apparatuses) aredisposed at predetermined positions above the moving apparatus 10, awashing system in which washing nozzles (predetermined apparatuses) aredisposed at predetermined positions at sides of the moving apparatus 10,and so forth.

In the case of an inspection system, for example, an inspection sensor(predetermined apparatus) is disposed above the moving apparatus 10instead of the robot 30, and the controller 40 performs image processingand determination by using detection images acquired by the inspectionsensor. An inspection illumination apparatus for illuminating thesensing area of the inspection sensor may be provided.

Also, the controller 40 creates AR images of the image processingresults, determination results, parameters related thereto, scorescorresponding to the parameters, and so forth, and transmits the createdAR images to the display device 62. For example, the display device 62displays AR images of image-processed images of inspection targetobjects, determination results related to pass/fail of the inspectiontarget objects, parameters related thereto, scores, and so forth. Inaddition, the display device 62 may display noise images, which areimage-processed images of objects other than the inspection targetobjects, patterns, blemishes, and so forth on the moving apparatus 10,parameters related to the noise images, scores of the noise images, andso forth. With this configuration, it becomes possible for the operatorto easily perform the setting of the controller 40, the setting work ofthe inspection illumination apparatus, and so forth.

As has been described above, with this embodiment, the AR images 71, 72,73, 75, 76, and 77, which show the settings related to the detections ofthe target objects O performed by using the detecting device 20 beingrecognized by the controller 40, the settings of the moving apparatus10, and the settings of the robot 30 that performs the work on thetarget objects O, are created to be displayed on the display device 62.Accordingly, it is possible for the operator to easily recognize theseitems of information.

In addition, with this embodiment, it is also possible for the operatorto visually recognize the positions of the target objects O beingrecognized by the controller 40 on the basis of the settings, theresults of the detections of the target objects O performed by using thedetecting device 20, the work plan of the moving apparatus 10, the workplan of the robot 30, the determination results of the controller 40related to the execution of the work performed by the robot 30, theparameters related to the target objects O being recognized by thecontroller 40, and so forth, and, by doing so, it is possible for theoperator to easily know what processing is actually being performed inthe controller 40.

Visually recognizing the settings, the processing, and so forth in thecontroller 40 in this way facilitates investigation of causes ofsuccess/failure of the operation of the system.

In addition, in this embodiment, the controller 40 updates, when therobot 30 is performing work, the AR images 73, 78, 79, 80, and 81 thatshow at least one type of information among the positions of the targetobjects O being recognized by the controller 40, the results of thedetections of the target objects O, the work plan of the movingapparatus 10 that the controller 40 receives or sets, the work plan ofthe robot 30 set by the controller 40, the determination results of thecontroller 40, and the parameters related to the target objects O beingrecognized by the controller 40.

Because of this, it is possible for the operator to visually recognizedynamic changes in the settings, processing, and so forth of thecontroller 40 related to these items of information.

In addition, in this embodiment, the controller 40 controls the robot30; the selecting processing, in which at least one of the AR imagesdisplayed on the display device 62 in accordance with the input made byusing the predetermined input device is selected, and the movementprocessing, in which the selected AR image is moved on the displaydevice 62 in accordance with the input made by using the predeterminedinput device, are performed; and the parameters that are set orcalculated in the controller 40 to control the robot 30 are changed inaccordance with the movement information of the selected AR image.

Because of this, it becomes possible for the operator to intuitivelychange the settings of the controller 40 for controlling the robot 30,which is useful, also for a skilled operator, for enhancing the settingaccuracy and the work efficiency.

In addition, in this embodiment, the controller 40 creates the pluralityof AR images, and the display device 62 is capable of displaying the ARimages selected from among the plurality of AR images in accordance withthe inputs to the predetermined input unit.

By doing so, it is possible for the operator to select the AR images tobe displayed via the inputs to the input unit. Accordingly, it becomespossible for the operator to more easily recognize the processing,settings, and so forth of the controller 40, and it is extremely usefulfor facilitating the investigation of causes of success/failure of theoperation of the system.

In addition, this embodiment is provided with the storage units 43 and63 that store the AR images created by the controller 40 together withthe corresponding real-space images.

By doing so, it is possible for the operator to replay the stored ARimages together with the real-space images, which is extremely usefulfor facilitating checking of the settings of the system, facilitatingsetting of the system, and facilitating investigation of causes ofsuccess/failure of the operation of the system. Note that replaying,slow replaying, pausing display during a replay, and so forth may beperformed only for the saved AR image data. Such a configuration alsocontributes to facilitating checking of the settings of the system,facilitating setting of the system, and facilitating investigation ofcauses of success/failure of the operation of the system.

Note that the robot 30, which is an articulated robot, is employed inthis embodiment as a work machine. In contrast, it is also possible toemploy, as a work machine, a drill-supporting apparatus that moves atool such as a drill only in top-to-bottom directions by means of apneumatic cylinder, a hydraulic cylinder, or the like, and it is alsopossible to employ other types of work machine.

In addition, it is also possible to employ a display device thatprojects AR images onto the retina of the operator instead of thedisplay device 62. In this case also, the AR images are displayed on theretina of the operator together with the real-space images.

In addition, it is also possible to use, instead of display by means ofthe display device 62 or together with display by means of the displaydevice 62, a projector (projection apparatus) that projects AR imagesonto at least one of apparatuses, for example, the moving apparatus 10,the detecting device 20, and the robot 30, and the spaces in real space.Because the AR images are also displayed in real space in this case, itis possible to achieve operational effects that are similar to thosedescribed above.

With the projection apparatus, it is also possible to project 3D ARimages in real space. For example, by projecting AR images in a workregion, it is possible to display information in the work region.

In addition, unlike the case described above, the glasses with an ARfunction, the head-mounted display, or the like, which is the displaydevice 62, may be provided with a light-transmitting plate or alight-transmitting film that allows the operator wearing such a deviceto view raw real space. In this case, the light-transmitting plate orthe light-transmitting film may be provided with a function fordisplaying AR image data, and the glasses with an AR function, thehead-mounted display, or the like may be provided with a function forprojecting AR image data onto the light-transmitting plate or thelight-transmitting film. By employing said configuration, because it ispossible for the operator to view the AR images on thelight-transmitting plate or the light-transmitting film while viewingreal space via the display device 62, operational effects that aresimilar to those described above are achieved.

Regarding positioning between the augmented-reality images on thelight-transmitting plate or the light-transmitting film and raw realspace, for example, it is possible to employ a method in which theposition at which a person is standing or the like is detected by usinganother sensor and AR images in accordance with the orientation of theperson are prepared and displayed.

In the case in which the display device 62 or the projection apparatusis employed, with which it is possible to view the AR images on thelight-transmitting plate or the light-transmitting film while viewingreal space, it is also possible to perform the above-described“Displaying detection results that do not qualify as work target”, theabove-described “Displaying augmented-reality image in accordance withselection”, and the above-described “Selection and manipulation ofobject”, and the application to another work system or an inspectionsystem is also possible, as described above.

For example, when a motion of grabbing a projected AR image or an ARimage displayed on the light-transmitting plate or thelight-transmitting film with the index finger and the thumb of a smartglove is performed, said motion is recognized by means of an imageacquired by the camera, and said AR image is moved when the index fingerand the thumb are moved while being held in that state.

Note that the camera 61 of the augmented reality apparatus 60 may bedisposed at a position from which the moving apparatus 10, the detectingdevice 20, and the robot 30 are overlooked, and the display device 62may display real-space images and AR images in an overlooking state. Inthis case, the display device 62 may display an AR image of theoperator.

Note that, in this embodiment, although a system in which the controller40 creates AR images, as described above, has been described, acontroller that is built into the augmented reality apparatus 60 or thelike or another controller may receive data about various settings, thecomputation results, the determination results, and so forth from thecontroller 40, may create AR images by using the received data, and maytransmit the created AR image to the display device 62.

In addition, the detecting device 20 may move in accordance with thepositions of the target objects O. For example, the detecting device 20may be supported at a tip portion of the robot 30, and, in this casealso, the controller 40 has setting data such as the coordinate system,the detection area, and so forth of the detecting device 20, and iscapable of determining, by means of calculations, the position andorientation of the detecting device 20 in accordance with the motions ofthe robot 30.

Furthermore, the detecting device 20 may be supported by a single-axisrobot, an articulated robot, or the like. In such a case, the controller40 receives the setting data such as the coordinate system, thedetection area, and so forth of the detecting device 20, and is capableof determining, by means of calculations, the position and orientationof the detecting device 20 in accordance with the motions of thesingle-axis robot, the articulated robot, or the like.

Note that the augmented-reality creating unit need not be in thecontroller.

The following aspects of the present invention are derived from theabove disclosure.

A sensing system according to a first aspect of the present invention isprovided with: a detecting device that is used to detect at least aposition of a target that is being moved by a moving means for movingthe target; and a controller, wherein the controller creates, fordisplay on a display device or projection by means of a projectionapparatus, an augmented-reality image that shows at least one type ofthe following information: at least one of a setting related todetection of the target performed by using the detecting device, asetting of the moving means, a setting of a work machine that performswork on the target, which are settings being recognized by thecontroller; a position of the target being recognized by the controllerbased one basis of the settings; a result of the detection of the targetperformed by using the detecting device, a work plan of the movingmeans, a work plan of the work machine, a determination result of thecontroller related to execution of the work performed by the workmachine, and a parameter related to the target being recognized by thecontroller.

Hereinafter, there are cases in which the “augmented-reality image” isdescribed as an “AR image”.

With this aspect, the AR images of the settings related to the detectionof the target performed by using the detecting device, the setting ofthe moving means, the setting of the work machine that performs the workon the target, and so forth, which are being recognized by thecontroller, are created in order to be displayed by a display device orprojected by a projection apparatus. For example, an AR image that showsthe detection region of the detecting device, an AR image that shows thetransfer direction of the moving means, an AR image that shows the workregion of the work machine, and so forth are created. Accordingly, it ispossible for an operator to easily recognize these items of information.

In addition, with this aspect, it is also possible for the operator tovisually recognize the position of the target being recognized by thecontroller based on the settings, the result of the detection of thetarget performed by using the detecting device, the work plan of themoving means, the work plan of the work machine, the determinationresult of the controller related to execution of the work performed bythe work machine, the parameter related to the target being recognizedby the controller, and so forth, and, accordingly, it is possible forthe operator to easily know what processing is actually being performedin the controller.

Visually recognizing the settings, the processing, and so forth in thecontroller in this way facilitates investigation of causes ofsuccess/failure of the operation of the system.

In the above-described aspect, preferably, the projection apparatusprojects the augmented-reality image onto real space, or the displaydevice displays the augmented-reality image so that theaugmented-reality image can be viewed together with a real-space imageor a real space.

In the case in which the AR images are projected onto real space, it ispossible for the operator to visually recognize the settings,processing, and so forth in the controller in real space, which issuitable for holding a discussion among multiple persons. In the case inwhich the AR images are displayed on the display device so that the ARimages can be viewed together with the real-space images or real space,it is possible for the operator to visually recognize the relationshipbetween the AR images and real space by simply viewing the display onthe display device without having to view the entire system. Inaddition, in the case in which the display device displays the AR imagestogether with the real-space images, it is possible to ascertain thesituation even if the operator is not at the work site, which isparticularly suitable for system monitoring. Adding the AR images toreal space is particularly suitable in the case in which the work isperformed while actually viewing the work site.

In the above-described aspect, preferably, the controller updates, whenthe work machine is performing the work, the augmented-reality imagethat shows at least one type of the following information: the positionof the target being recognized by the controller; the result of thedetection of the target; the work plan of the moving means; the workplan of the work machine; the determination result of the controller;and the parameter related to the target.

With this aspect, because the AR images are updated in order to bedisplayed by a display device or projected by a projection apparatus, itis possible for the operator to visually recognize dynamic changes ofthe settings, processing, and so forth of the controller related to theAR images.

In the above-described aspect, preferably, wherein the controllercontrols the work machine, the sensing system further includes: anaugmented-reality-image manipulating unit which performs selectingprocessing that selects a least one of the augmented-reality imagesdisplayed on the display device or projected by the projection apparatusin accordance with an input made by using a predetermined input device,the augmented-reality-image manipulating unit which performs movementprocessing that moves the selected augmented-reality image displayed onthe display device or projected by the projection apparatus inaccordance with the input made by using the predetermined input device;and a setting adjusting unit which changes, in accordance with movementinformation of the selected augmented-reality image, a parameter that isset or calculated in the controller to control the work machine.

With this aspect, the operator moves the AR images by using thepredetermined input device, and the setting adjusting means changes theparameters that are set or calculated in the controller. Accordingly, itbecomes possible for the operator to intuitively change the settings forcontrolling the work machine, which is useful, also for a skilledoperator, for enhancing the setting accuracy and the work efficiency.

In the above-described aspect, preferably, wherein the controllercreates a plurality of the augmented-reality images, wherein the displaydevice or the projection apparatus displays or projects anaugmented-reality image selected from the plurality of augmented-realityimages in accordance with an input to a predetermined input unit.

Alternatively, the controller creates and records the augmented-realityimages in real time, and, displays the augmented-reality image selectedfrom the plurality of augmented-reality images at that point in time ora plurality of augmented-reality images from the immediate past, inaccordance with the input to the predetermined input unit orestablishment of predetermined conditions in the controller or thesensor.

With this aspect, it is possible for the operator to select the AR imageto be displayed by using the input to the input unit or the conditionsetting with respect to the controller or the sensor. Accordingly, itbecomes possible for the operator to more easily recognize theprocessing, settings, and so forth of the controller at an arbitrarypoint in time or at the time of phenomenon occurrence, which isextremely useful for facilitating investigation of causes ofsuccess/failure of the operation of the system and enhancing thecapacity utilization of the system.

The above-described aspect is provided with, preferably, a storage unitthat stores the augmented-reality image created by the controllertogether with a corresponding real-space image.

With this aspect, the AR images are stored together with thecorresponding real-space images. Accordingly, it is possible for theoperator to replay the stored AR images together with the real-spaceimages, which is extremely advantageous for facilitating checking of thesettings of the system, facilitating setting of the system, andfacilitating investigation of causes of success/failure of the operationof the system.

The above-described aspect is provided with, preferably, a replayingunit which replays the augmented-reality image stored in the storageunit or replays the augmented-reality image together with the real-spaceimage.

With this aspect, it is possible for the operator to easily replay, bymeans of the replaying means, the stored AR images at an arbitrarylocation together with the real-space images.

Alternatively, it is possible to perform, on-site, adjustments forenhancing the capacity utilization by replaying the augmented-realityimages while viewing the apparatuses in the real spaces.

A sensing system according to a second aspect of the present inventionis provided with: a detecting device that is used to detect at least aposition of a target that is being moved by a predetermined movingmeans; a controller that controls a work machine that performs work onthe target by using detection result of the detecting device; anaugmented-reality-image creating unit which successively creates, whenthe work machine is performing the work, augmented-reality images thatshow at least one type of the following information: the position of thetarget being recognized by the controller on the basis of the detectionresult of the detecting device, a work plan of the moving means, a workplan of the work machine, a determination result of the controllerrelated to execution of the work performed by the work machine, and aparameter related to the target being recognized by the controller; anda storage unit that stores the augmented-reality images created by theaugmented-reality-image creating unit together with correspondingreal-space images.

With this aspect, the AR images are stored together with thecorresponding real-space images. Accordingly, it is possible for theoperator to replay the stored AR images together with the real-spaceimages, which is extremely advantageous for facilitating checking of thesettings of the system, facilitating setting of the system, andfacilitating investigation of causes of success/failure of the operationof the system.

The above-described aspect is provided with, preferably, a replayingunit which replays the augmented-reality images stored in the storageunit or replays the augmented-reality images together with thereal-space images.

A sensing system according to a third aspect of the present invention isprovided with: a detecting device that is used to detect a plurality oftargets that are being moved by a predetermined moving means; a targetdetecting unit which applies image processing for detecting the targetsto the image data obtained by the detecting device; anaugmented-reality-image creating unit which creates for a candidate in aplurality of candidates that appear in the processed image to which theimage processing has been applied, an augmented-reality image thatcorresponds to the candidate in which a predetermined parameter is equalto or greater than a detection threshold, and creates a candidateaugmented-reality image that corresponds to a candidate in the pluralityof candidates, in which the predetermined parameter is less than thedetection threshold and equal to or greater than a detection candidatethreshold; and a display device or a projection apparatus that displaysor projects the augmented-reality image and the candidateaugmented-reality image.

With this aspect, not only the AR image that corresponds to thecandidate in which the predetermined parameter is equal to or greaterthan the detection threshold, but also the candidate AR image thatcorresponds to the candidate in which, although the predeterminedparameter is less than the detection threshold, the predeterminedparameter is equal to or greater than the detection candidate thresholdis displayed on the display device in a distinguishable form, such as adifferent color. Accordingly, the work for increasing the number ofdetected targets through a change in the detection threshold, a changein lighting (illumination) for detection, changes in other settings, andso forth is facilitated, which also facilitates checking of the resultsafter performing said work.

In the above-described aspect, preferably, the candidateaugmented-reality image is created in a form that is distinguishablefrom the augmented-reality image.

With this aspect, it is possible for the operator to easily recognizethe presence of the target corresponding to the candidate AR image.

Note that the created augmented-reality image may be saved in a storageapparatus.

In the above-described aspect, preferably, the augmented-reality-imagecreating unit creates an augmented-reality image of a detection resultaccompanying the candidate augmented-reality image, and the displaydevice or the projection apparatus displays or projects theaugmented-reality image of the detection result together with thecandidate augmented-reality image.

With this aspect, it becomes possible for the operator to more easilyset parameters related to the detecting device on the basis of thedetection result that is displayed together with the candidate AR image.

A work system according to a fourth aspect of the present invention isprovided with the sensing system and a work machine.

A fifth aspect of the present invention is an augmented-reality-imagedisplaying method that is employed in a system in which a controllercontrols a work machine or a predetermined apparatus based on adetection result of a target that is being moved by a moving means formoving the target, the augmented-reality-image displaying methodincluding: creating by the controller an augmented-reality image thatshows at least one type of the following information: at least one of asetting related to detection of the target, a setting of the movingmeans, a setting of a work machine, which are settings being recognizedby the controller; a position of the target being recognized by thecontroller based on the settings; a result of the detection of thetarget performed by using the detecting device; a work plan of themoving means; a work plan of the work machine; a determination result ofthe controller related to execution of the work performed by the workmachine; and a parameter related to the target being recognized by thecontroller, and the method further including: displaying the createdaugmented-reality image on a display device or projecting theaugmented-reality image using a projection apparatus.

A sixth aspect of the present invention is an augmented-reality-imagestoring method that is employed in a system in which a controllercontrols a work machine based on a detection result of a target that isbeing moved by a moving means for moving the target, theaugmented-reality-image storing method including: successively creating,when the work machine is performing the work, augmented-reality imagesthat show at least one type of the following information: a position ofthe target being recognized by the controller based on the detectionresult; a work plan of the moving means; a work plan of the workmachine; a determination result of the controller related to executionof the work performed by the work machine; and a parameter related tothe target being recognized by the controller, and the method furtherincluding: storing the augmented-reality images created in theaugmented-reality-image creating step together with correspondingreal-space images.

A seventh aspect of the present invention is an augmented-reality-imagedisplaying method that is employed in a system that performs imageprocessing for detecting a plurality of targets that are being moved bya moving means for moving the targets, the augmented-reality-imagedisplaying method including: creating, for a candidate in a plurality ofcandidates that appear in the processed image to which the imageprocessing has been applied, an augmented-reality image that correspondsto the candidate in which a predetermined parameter is equal to orgreater than a detection threshold, and creating a candidateaugmented-reality image that corresponds to a candidate in the pluralityof candidates, in which the predetermined parameter is less than thedetection threshold and equal to or greater than a detection candidatethreshold; and displaying the augmented-reality image and the candidateaugmented-reality image.

An eighth aspect of the present invention is a program that is employedin a system in which a controller controls a work machine or apredetermined apparatus based on a detection result of a target that isbeing moved by a moving means for moving the target, the storage mediumstoring a program which causes a computer to execute: creating anaugmented-reality image that shows at least one type of the followinginformation: at least one of a setting related to the detection of thetarget, a setting of the moving means, a setting of a work machine,which are settings being recognized by the controller; a position of thetarget being recognized by the controller based on the settings; aresult of the detection of the target performed by using the detectingdevice; a work plan of the moving means; a work plan of the workmachine; a determination result of the controller related to executionof the work performed by the work machine; and a parameter related tothe target being recognized by the controller; and displaying thecreated augmented-reality image on a display device or projecting theaugmented-reality image using a projection apparatus.

A ninth aspect of the present invention is a program that is employed ina system in which a controller controls a work machine on the basis of aresult of detection of a target that is being moved by a moving meansfor moving the target, the storage medium storing a program which causesa computer to execute: successively creating, when the work machine isperforming the work, augmented-reality images that show at least onetype of the following information: a position of the target beingrecognized by the controller based on the detection result; a work planof the moving means; a work plan of the work machine; a determinationresult of the controller related to execution of the work performed bythe work machine; and a parameter related to the target being recognizedby the controller; and storing the augmented-reality images created inthe augmented-reality-image creating step together with correspondingreal-space images.

A tenth aspect of the present invention is a program that is employed ina system that performs image processing for detecting a plurality oftargets that are being moved by a moving means for moving the targets,the storage medium storing a program which causes a computer to execute:creating, for a candidate in a plurality of candidates that appear inthe processed image to which the image processing has been applied, anaugmented-reality image that corresponds to the candidate in which apredetermined parameter is equal to or greater than a detectionthreshold, and creating a candidate augmented-reality image thatcorresponds to a candidate in the plurality of candidates, in which thepredetermined parameter is less than the detection threshold and equalto or greater than a detection candidate threshold; and a displayingstep of displaying the augmented-reality image and the candidateaugmented-reality image.

The above aspects afford an advantage in that it is possible tofacilitate investigation of causes of success/failure of the operationof a system that detects a target, to facilitate checking of the settingof the system or to facilitate setting of the system, to enhance theefficiency of state change recognition, and to simplify systemadjustment.

REFERENCE SIGNS LIST

-   10 moving apparatus-   11 motor-   12 operating-position detecting device-   20 detecting device-   30 robot-   30 a arm-   31 servomotor-   40 controller-   41 control unit-   42 controller display unit-   43 storage unit-   43 a system program-   43 b motion program-   43 c object detecting program-   43 d AR-image creating program-   43 e replaying program-   50 tool-   51 servomotor-   60 augmented reality apparatus-   61 camera-   62 display device-   63 storage unit-   63 a AR-image manipulating program-   71-81 AR image-   O target object

1. A sensing system comprising: a detecting device that is used todetect a plurality of targets that are being moved by a predeterminedmoving means; a target detecting unit which applies image processing fordetecting the targets to the image data obtained by the detectingdevice; an augmented-reality-image creating unit which creates for acandidate in a plurality of candidates that appear in a processed imageto which the image processing has been applied, an augmented-realityimage that corresponds to the candidate in which a predeterminedparameter is equal to or greater than a detection threshold, and createsa candidate augmented-reality image that corresponds to a candidate inthe plurality of candidates, in which the predetermined parameter isless than the detection threshold and equal to or greater than adetection candidate threshold; and a display device or a projectionapparatus that displays or projects the augmented-reality image and thecandidate augmented-reality image.
 2. The sensing system according toclaim 1, wherein the candidate augmented-reality image is created in aform that is distinguishable from the augmented-reality image.
 3. Thesensing system according to claim 1, wherein the augmented-reality-imagecreating unit creates an augmented-reality image of a detection resultaccompanying the candidate augmented-reality image, and the displaydevice or the projection apparatus displays or projects, respectively,the augmented-reality image of the detection result together with thecandidate augmented-reality image.
 4. An augmented-reality-imagedisplaying method that is employed in a system that performs imageprocessing for detecting a plurality of targets that are being moved bya moving means for moving the targets, the augmented-reality-imagedisplaying method comprising: creating, for a candidate in a pluralityof candidates that appear in the processed image to which the imageprocessing has been applied, an augmented-reality image that correspondsto the candidate in which a predetermined parameter is equal to orgreater than a detection threshold, and creating a candidateaugmented-reality image that corresponds to a candidate in the pluralityof candidates, in which the predetermined parameter is less than thedetection threshold and equal to or greater than a detection candidatethreshold; and displaying the augmented-reality image and the candidateaugmented-reality image.
 5. A storage medium that stores a program thatis employed in a system that performs image processing for detecting aplurality of targets that are being moved by a moving means for movingthe targets, the storage medium storing a program which causes acomputer to execute: creating, for a candidate in a plurality ofcandidates that appear in the processed image to which the imageprocessing has been applied, an augmented-reality image that correspondsto the candidate in which a predetermined parameter is equal to orgreater than a detection threshold, and creating a candidateaugmented-reality image that corresponds to a candidate in the pluralityof candidates, in which the predetermined parameter is less than thedetection threshold and equal to or greater than a detection candidatethreshold; and a displaying step of displaying the augmented-realityimage and the candidate augmented-reality image.